Skip to content

String.split/1 refactor - #15781

Closed
dkuku wants to merge 1 commit into
elixir-lang:mainfrom
dkuku:dk_string_split_without_pattern_compilation
Closed

String.split/1 refactor#15781
dkuku wants to merge 1 commit into
elixir-lang:mainfrom
dkuku:dk_string_split_without_pattern_compilation

Conversation

@dkuku

@dkuku dkuku commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

Currently String.split/1 is using :binary.split(string, unquote(whitespace -- non_breakable), [:global, :trim_all]), which is only beneficial on long inputs like files 100kB+ — where you probably want to tweak your split chars anyway.
This pays a pattern-compilation cost on every call: the needle list is passed as a literal, so the automaton for the whitespace codepoints is rebuilt each time.

Refactor into multiple function heads — just like the other functions in this module.
I compared on a full file input (145K) where speed is comparable, the same file split by lines, and 2 word pairs.
The other 2 benchmarks are comparing splitting lines and 2 word pairs.
I know that comparing this in an Enum loop may show the existing implementation much slower but I think it is the most common pattern.

prop_path = "#{unicode_folder}/PropList.txt"
prop = File.read!(prop_path)
lines = String.split(prop, "\n")
words = String.split(prop) |> Enum.chunk_every(2) |> Enum.map(&Enum.join(&1," "))

Benchee.run(
  %{
    "binary_split" => fn input -> Enum.map(input, &String.Break.split/1) end,
    "offset - proposed" => fn input -> Enum.map(input, &String.BreakNew.split/1) end,

  },
  inputs: [
    {"words", words},
    {"lines", lines},
    {"file", [prop]}
  ],
  time: 1,
  memory_time: 1
) 
##### With input words #####
Name                        ips        average  deviation         median         99th %
offset - proposed        1.02 K        0.98 ms    ±27.56%        0.93 ms        1.70 ms
binary_split           0.0355 K       28.18 ms     ±4.21%       28.56 ms       30.86 ms

Comparison: 
offset - proposed        1.02 K
binary_split           0.0355 K - 28.69x slower +27.20 ms

Memory usage statistics:

Name                 Memory usage
offset - proposed       812.03 KB
binary_split            927.26 KB - 1.14x memory usage +115.23 KB

**All measurements for memory usage were the same**

##### With input lines #####
Name                        ips        average  deviation         median         99th %
offset - proposed        1.39 K        0.72 ms    ±14.94%        0.76 ms        0.94 ms
binary_split            0.162 K        6.18 ms     ±5.65%        6.14 ms        7.66 ms

Comparison: 
offset - proposed        1.39 K
binary_split            0.162 K - 8.57x slower +5.46 ms

Memory usage statistics:

Name                 Memory usage
offset - proposed       402.42 KB
binary_split            788.15 KB - 1.96x memory usage +385.73 KB

**All measurements for memory usage were the same**

##### With input file #####
Name                        ips        average  deviation         median         99th %
binary_split             1.20 K      830.18 μs    ±30.29%      738.89 μs     1748.19 μs
offset - proposed        1.11 K      898.37 μs    ±17.23%      975.08 μs     1229.29 μs

Comparison: 
binary_split             1.20 K
offset - proposed        1.11 K - 1.08x slower +68.19 μs

Memory usage statistics:

Name                 Memory usage
binary_split              1.27 KB
offset - proposed       296.11 KB - 232.53x memory usage +294.84 KB

**All measurements for memory usage were the same**

@josevalim

Copy link
Copy Markdown
Member

I pushed some PRs to OTP that make binary:split much faster because it uses SIMD, so I would like to see where those land first:

erlang/otp#11467
erlang/otp#11468

Thank you!

@josevalim josevalim closed this Aug 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants